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training  were  not  considered  essential  to  the  job  and,  therefore,  not  used. 
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to  practice  maintenance  skills,  and  this  sets  the  stage  for  the  deterioration 
of  these  skills. 
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FOREWORD 


Tills  Advanced  Development  effort  was  conducted  in  support  of  Project  Z0108-PN 
(ZPN07),  Education  and  Training  Development,  under  the  sponsorship  of  the  Chief 
of  Naval  Operations  (OP-099).  This  is  the  third  in  a series  of  reports  relating 
to  Subproject  Z0108-PN.24,  Personnel  Readiness  Training.  The  first  report,  NPKDC 
Special  Report  75-8,  A Personnel  Readiness  Training  Program:  Initial  Project 

Developments , provided  an  overview  of  the  project.  The  second  report,  NPRDC 
Technical  Report  77-4,  A Personnel  Readiness  Training  Program:  Operation  of 

the  AN/BQR-20A,  described  the  program  for  the  submarine  Sonar  Technician. 

A subsequent  report  will  describe  the  program  for  the  Boiler  Technician,  and 
a final  report  will  summarize  findings  and  conclusions  across  all  three  appli- 
cations . 

The  diagnostic  tests  and  shipboard  training  materials  for  the  Missile 
Technician  program,  which  are  described  in  this  report,  were  developed  under 
contract  N00123-74-C1579,  awarded  to  Data  Design  Laboratories,  Cucamonga, 
California. 

Appreciation  is  expressed  to  the  following: 

• Commander  Submarine  Forces  Atlantic  and  Commander  Strategic  Systems  Project 
Office,  Training  Systems  Branch,  for  cooperation  and  support  in  carrying  out 

this  research. 

• Commander  Submarine  School,  New  London  and  Commander  Fleet  Ballistic 
Missile  Submarine  Training  Center,  Charleston  for  providing  facilities  and 
assistance  in  testing. 

• Commanding  Officers  and  Missile  Technicians  of  the  12  submarines  who 
participated  in  the  testing  program. 

Special  appreciation  is  expressed  to  the  Officer  in  Charge  of  the  Central 
Test  Site  for  the  Personnel  and  Training  Evaluation  Program,  the  Directors  of 
the  Central  Test  Site  Detachments  in  New  London  and  Charleston,  and  their 
respective  staffs  for  outstanding  assistance  in  scheduling  submarine  crews  and 
collecting  the  data.  Also,  the  valuable  assistance  of  STCS(SS)  John  C.  Myers 
and  Hervey  Stern  in  conducting  Fleet  testing  is  gratefully  acknowledged. 
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SUMMARY 


Problem 


The  Personnel  Readiness  Training  Program  is  concerned  with  the  feasibility 
of  using  a diagnostic  testing/ shipboard  training  system  to  improve  the  readiness 
levels  of  Fleet  personnel.  In  such  a system,  performance-oriented  tests  are 
used  to  diagnose  deficiencies  in  job  performance  and  shipboard  self-instructional 
materials  are  individually  prescribed  to  correct  deficiencies  revealed  by  the 
diagnostic  tests.  To  obtain  information  on  how  and  where  this  type  of  system 
might  work,  testing  and  training  programs  were  developed  for  three  applications: 

(1)  the  submarine  Sonar  Technician  (ST)  operating  the  AN/BQR-20A,  (2)  the 
submarine  Missile  Technician  (MT)  maintaining  the  Missile  Test  and  Readiness 
Equipment  (MTRE  Mk  7 Mod  2),  and  (3)  the  Boiler  Technician  (BT)  operating  and 
maintaining  the  1200  PSI  Steam  Propulsion  Plant. 

Purpose 

The  purpose  of  the  effort  described  in  this  report  was  to  determine  whether 
the  testing  and  training  programs  developed  for  the  MT  application  were  instru- 
mental in  improving  maintenance  performance. 

Approach 

Missile  Technicians  from  12  Fleet  Ballistic  Missile  (FBM)  submarines  were  given 
diagnostic  pretests  and  then  retested  approximately  5 months  later.  The  tests 
covered  MTRE  troubleshooting.  Standard  Maintenance  Procedures,  and  the  operation 
of  test  equipments  used  in  MTRE  maintenance.  Three  experimental  groups,  each 
consisting  of  the  MTs  from  four  submarines,  were  used  in  the  evaluation:  a 

Control  Group,  a Diagnostic  Feedback  Group,  and  a Diagnostic  Feedback  + Training 
Group.  Following  the  pretests,  members  of  the  Control  Group  were  given  feed- 
back on  their  performance  in  terms  of  an  overall  percentage  score.  Members  of 
the  Feedback  Group  were  provided  with  a list  of  their  individual  areas  of  weak- 
ness. Members  of  the  Diagnostic  Feedback  + Training  Group  were  provided  with 
the  same  type  of  information  as  the  Feedback  Group,  but  they  were  also  assigned 
specific  remedial  training  materials  that  covered  their  individual  weaknesses. 

Findings 

Diagnostic  testing  was  successful  in  detecting  deficiencies  of  Fleet 
personnel  on  tasks  related  to  the  maintenance  of  MTRE.  Those  MTs  who  had 
completed  the  MTRE  Maintenance  Course  and/or  appropriate  test  equipment  course 
had  fewer  deficiencies.  Neither  feedback  in  the  form  of  specific  performance 
deficiences  nor  feedback  plus  specifically  assigned  remedial  training  improved 
job  performance.  Apparently,  one  of  the  reasons  the  MTs  showed  deficiencies 
is  that  they  do  not  get  to  practice  maintenance  skills  because  MTRE  is  exception- 
ally reliable.  The  reliability  of  MTRE  may  also  be  one  of  the  reasons  that  the 
feedback  and/or  remedial  training  were  not  perceived  as  essential  to  the  job. 

The  following  additional  factors  may  have  further  contributed  to  the  nonuse  of 
the  feedback  and/or  remedial  training: 

1.  On  a given  patrol,  only  one  MT  has  responsibility  for  MTRE  and  this  assign- 
ment usually  occurs  only  once  during  his  tour  on  an  FBM  submarine. 


vii 


PRfiCEDINO  PAGE  BLANK*- NOT  FILMED 


2.  The  training  packages  for  most  MTs  were  relatively  large  and  the  individual 
lessons  were  relatively  long. 

Conclusions 


There  are  serious  deficiencies  in  tasks  related  to  the  maintenance  of  MTRE, 
but  an  MT  will  not  be  sufficiently  motivated  to  study  remedial  training  materials 
unless  he  is  about  to  be  given  responsibility  for  MTRE.  Appropriate  formal 
courses  and  remedial  training,  if  needed,  should  remedy  the  deficiencies. 

Recommendations 


To  improve  the  proficiency  level  of  the  MT  designated  to  have  responsibility 
for  operating  and  maintaining  MTRE,  it  is  recommended  that: 

1.  If  the  designated  MT  has  not  already  attended  the  MTRE  maintenance  and 
test  equipment  courses,  he  should  attend  these  courses  during  the  off-crew  period 
immediately  preceding  the  patrol  on  which  he  will  assume  MTRE  responsibility. 

2.  If  the  designated  MT  has  attended  the  MTRE  maintenance  and  test  equipment 
courses,  he  should  be  given  the  MTRE  diagnostic  test  at  the  beginning  ol'  the 
off-crew  period  immediately  preceding  the  patrol  on  which  tie  will  assume  MTRE 
responsibility.  He  should  then  be  assigned  the  appropriate  remedial  training 
packages  and  be  retested  at  the  completion  of  training. 
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INTRODUCTION 


Problem 

The  Personnel  Readiness  Training  Program  is  concerned  with  the  feasibility 
of  using  a diagnostic  testing/shipboard  training  system  to  improve  the  readi- 
ness levels  of  Fleet  personnel.  In  such  a system,  performance-oriented  tests 
are  used  to  diagnose  deficiencies  in  job  performance  and  shipboard  self-instruc- 
tional materials  are  individually  prescribed  to  correct  deficiencies  revealed 
by  the  diagnostic,  tests. 

Background 

The  degree  to  which  critical  job  skills  can  be  improved  through  a system  of 
diagnostic  testing  and  shipboard  training  may  depend  on  the  rating  and/or  the 
type  of  task  involved.  Therefore,  to  obtain  information  on  how  and  where  the 
system  might  work,  testing  and  training  programs  were  developed  for  three  applica- 
tions. The  ratings  and  related  equipments  selected  for  the  program  evaluation 
were:  (1)  the  submarine  Sonar  Technician  (ST)  operating  the  AN/BQR-20A,  (2) 

the  submarine  Missile  Technician  (MT)  maintaining  the  Missile  Test  and  Readiness 
Equipment  (MTRE  Mk  7 Mod  2),  and  (3)  the  Boiler  Technician  (BT)  operating  and 
maintaining  the  1200  PSI  Steam  Propulsion  Plant.  These  ratings  and  equipments 
were  chosen  because  they  were  critical  to  the  missions  of  their  ships  and  they 
contained  a sufficient  variety  of  operator  and  maintenance  tasks  to  permit  the 
results  to  be  generalized  to  other  areas.  Additionally,  there  were  indications 
that  performance  deficiencies  might  be  present  in  these  areas. 

The  first  report  in  this  series  (Laabs,  Main,  Abrams,  & Steinemanr.,  Note  1) 
described  the  general  approach  and  how  it  was  being  applied  in  the  three  areas. 

The  second  report  (Winchell,  Panell,  & Pickering,  1976)  described  the  Personnel 
Readiness  Training  Program  for  the  ST.  A subsequent  report  will  describe  the 
program  for  the  BT;  a final  report  will  summarize  findings  and  conclusions  across 
all  three  applications. 

Purpose  and  Scope 

MTRE  is  used  onboard  Fleet  Ballistic  Missile  (FBM)  submarines  to  perform 
functional  tests  on  POSEIDON  missiles  and  to  verify  their  operational  readiness 
for  launch.  This  piece  of  equipment  requires  a single  operator  who,  in  practice, 
is  the  equipment's  sole  maintainer.  The  responsibility  for  the  maintenance  of 
MTRE,  which  is  assigned  to  a designated  MT  for  each  patrol,  carries  along  with 
it  a unique  problem  because  of  the  equipment's  exceptional  reliability.  Since 
MTRE  operates  without  failure  for  extended  periods  of  time,  MTs  have  little 
opportunity  to  practice  maintenance  skills.  Consequently,  their  MIRK  maintenance 
proficiency  can  be  expected  to  decline. 

The  purpose  of  the  present  effort  was  to  determine  whether  the  testing  and 
training  programs  developed  for  the  MT  were  instrumental  in  improving  performance 
on  tasks  related  to  the  maintenance  of  MTRE  Mk  7 Mod  2. 

Not  all  of  the  knowledges  and  skills  required  to  maintain  MTRE  could  he  tested 
within  acceptable  time  limits.  Therefore,  a selection  from  critical  tasks  was 
made,  anil  the  scope  of  the  program  was  limited  to  a reasonable  number  of  problems 
related  to  MTRE  troubleshooting,  Standard  Maintenance  Procedures,  anil  the  operation 
of  test  equipment  used  in  MTRE  maintenance.  Testing  was  accomplished  in  short - 
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based  training  facilities  with  the  actual  equipment  during  FBM  submarine  off-crew 
periods.  All  MTs  are  currently  tested  by  the  FBM  Personnel  and  Training  Evaluation 
Program  (PTF.P)  with  paper-and-pencil  tests  during  each  off-crew  period.  The 
performance  testing  of  the  present  program  was  integrated  with  this  existing 
evaluation  system. 
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METHOD 


The  diagnostic  tests  and  training  materials  were  developed  under  contract 
by  Data  Design  Laboratories,  Cucamonga,  California.  The  contract  effort  consisted 
of  the  following  four  phases: 

• Phase  I.  Survey  of  knowledges  and  skills  required  by  the  MT  who  maintains 
MTRE,  and  selection  of  critical  tasks  to  be  covered  by  a performance-oriented 
diagnostic  test. 

• Phase  II.  Development  of  a performance-oriented  diagnostic  test  to  provide 
information  on  distinct  and  separate  areas  of  potential  deficiency. 

• Phase  III.  Review  of  recent  developments  in  education  and  training,  and 
the  identification  of  instructional  formats  suitable  for  shipboard  training. 

• Phase  IV.  Development  of  shipboard-training  packages  that  correspond 
to  the  areas  of  potential  deficiency. 

Task  Selection 


Personnel  Performance  Profiles  (PPPs),  which  are  contained  in  Navy  Ordinance 
documents,  were  used  as  a guide  for  task  selection.  These  profiles  are  tables 
that  list  the  knowledges  and  skills  required  to  operate  and  maintain  a subsystem 
or  piece  of  equipment  of  the  FBM  Weapon  System.  In  effect,  each  table  is  a very 
general  task  analysis.  The  table  that  contains  the  PPPs  related  to  MTRE  mainte- 
nance (Note  2,  Table  310)  is  quite  extensive,  and  time  constraints  made  it  impossible 
to  test  all  items  with  hands-on  performance  tests.  Therefore,  Trouble  and  Failure 
Reports  were  examined  to  find  areas  suitable  for  the  development  of  hands-on 
performance  tests.  From  an  analysis  of  equipment  failures,  nine  critical  mal- 
functions were  identified.  Maintenance  procedures  were  selected  that  would  either 
lead  to  the  discovery  of  the  malfunctions  or  be  used  in  the  correction  of  the 
malfunctions.  The  maintenance  skill  and  knowledge  PPP  items  that  are  required 
to  complete  these  procedures  were  then  listed  and  used  in  constructing  a perfor- 
mance test.  Since  a performance  test  covering  all  nine  malfunctions  would  require 
approximately  10  hours,  only  three  malfunctions  were  used  as  the  basis  for  develop- 
ing a 1‘j-hour  preventive  maintenance  test.  Two  of  the  remaining  malfunctions 
were  used  as  the  basis  for  developing  a l^-hour  hands-on  troubleshooting  test. 

One  of  these  two  malfunctions  and  the  four  remaining  ones  were  used  as  the  basis 
for  developing  five  written  troubleshooting  problems.  The  time  needed  to  complete 
these  problems  ranged  from  about  20  to  90  minutes. 

To  minimize  the  demand  on  MTRE  equipment,  test  equipment  performance  testing 
was  divorced  from  the  MTRE  test  situation.  Instead,  separate  problems  using 
electronic  signals  from  a specially  designed  Test-Signal  Generator  were  devised 
for  the  three  pieces  of  test  equipment.  The  technical  manuals  for  the  test  equip- 
ment were  used  as  the  basis  for  developing  a 1-hour  Test  Equipment  Test.  The 
separation  of  test  equipment  use  from  other  aspects  of  MTRE  maintenance  performance 


Requests  for  information  concerning  the  informal  reports  for  Phase  T and 
III  and  the  end-products  for  Phase  II  and  IV  of  the  Data  Design  Laboratory  contract 
should  be  addressed  to  the  senior  author  of  the  present  report. 
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resulted  in  a test  equipment  t es t ing-and- 1 rain  1 up  program  that  could  be  used 
with  other  technical  ratings.  In  addition,  by  separating,  test  equipment  perfor- 
mance from  M'l'RE  maintenance  tasks,  potential  confounding  of  the  MTRE  Lest  situa- 
tion due  to  the  inability  of  Mis  to  operate  Lhe  test  equipment  was  eliminated. 

Diagnostic  Tests 

MTRE  Preventive  Maintenance  Test 


This  test  consisted  of  three  problems.  The  observer's  checksheet  for 
each  problem  listed  observable  performance  steps  that  replicated  steps  in  a 
Standard  Maintenance  Procedure  or  a step-by-step  procedural  guide  in  a maintenance 
manual.  The  first  problem  involved  a tape  fault  and  required  the  completion  of 
27  performance  steps  involving  tape  splicing  and  canister  reloading.  The  second 
problem  required  the  MT  to  complete  19  performance  steps  of  the  Digital  Multimeter 
Electrical  Adjustment  Check.  The  third  problem  involved  a potentiometer  which 
was  out-of-tolerance  and  required  the  completion  of  20  performance  steps  of  the 
Tape  Reader  Electrical  Check  and  Adjustment  Procedure.  Each  performance  step 
was  marked  as  either  correct  or  incorrect.  The  MT  was  supplied  with  all  documen- 
tation, tools,  and  test  equipments  necessary  to  solve  the  problem.  Prior 
to  each  test  session,  the  test  equipments  were  calibrated  and  completely  set-up 
by  the  observer. 

MTRE  Corrective  Maintenance  Test 


This  test  consisted  of  two  troubleshooting  problems  on  MTRE.  It  was 
administered  in  place  of  the  preventive  maintenance  problem  to  the  one  MT  on  each 
submarine  who  had  primary  responsibility  for  the  operation  and  maintenance  of 
MIRK  on  his  submarine's  last  patrol.  One  test  problem  used  a prefaulted  module 
and  was  assessed  as  being  moderately  difficult;  the  other  involved  the  niisadjust- 
ment  of  a potentiometer  on  an  amplifier  card  and  was  assessed  as  being  very 
difficult.  The  observer's  checksheet  for  these  problems  provided  fur  (1)  a count 
of  the  number  of  measurements  made  while  troubleshooting  along  with  an  indication 
of  whether  or  not  the  fault  had  been  corrected  and  (2)  the  observation  of  a 
limited  number  of  performance  steps  that  replicated  those  in  the  preventive 
maintenance  problems.  The  former  was  used  along  with  the  results  of  a simulated 
troubleshooting  test  to  assign  remedial  Instruction  related  to  corrective 
maintenance.  The  latter  was  used  to  assign  remedial  instruction  relative  to 
preventive  maintenance. 

Decision  Measurement  System  (DMS)  Test 

Tills  simulated  troubleshooting  test  was  comprised  of  five  problems  dosigne 
to  measure  Lhe  ability  of  the  MT  to  logically  apply  his  knowledge  of  equipment 
maintenance.  This  test  is  related  to  the  early  "Trainer- Test  or"  devices  developed 
by  Van  Valkonburgh , Nooger,  and  Neville,  Inc.  (e.g..  Notes  1 and  •'♦).  For  a given 
set  of  conditions,  the  "Trainer-Tester"  allows  free  exploration  of  test  points, 
pins,  contacts,  terminals  of  modules,  relays,  etc.  Thus,  multiple-path  solutions 
exist  when  using  this  device.  Itt  the  DMS  Test,  however,  only  four  selections 
are  allowed  for  each  major  step  in  a single-path  troubleshooting  sequence.  The 
DMS  Test  sheets  use  latent  Image  (Invisible  ink)  printing,  which  refers  to  the 
next  step  only  when  a correct  selection  is  made.  The  MT  continues  to  make 
selections  until  the  correct  one  is  made.  Performance  on  a problem  was  scored 
according  to  the  normal  PTEP  scoring  procedure  in  which  nine  points  are  assigned 


for  making  the  correct  selection  in  one  tr>  , eight  points  for  making  it  in  two 
tries,  six  points  for  making  it  in  three  tries,  and  zero  points  for  making  it 
on  the  fourth  try. 


Test  Equipment  Test 

This  test  consisted  of  eight  problems  on  three  different  pieces  of  test 
equipment.  The  specially  designed  Test-Signal  Generator  was  used  to  provide  the 
electronic  signals  for  the  test  problems.  Three  problems  on  the  AN/USM-281A 
oscilloscope  involved  initial  turn-on  procedures  and  calibrations,  periodic  signal 
analysis,  and  two-signal  phase  analysis.  Three  problems  involved  the  use  of  the 
John  Fluke  803D/AG  differential  voltmeter  to  measure  negative  DC  voltage,  AC 
voltage,  and,  as  a conventional  vacuum  tube  voltmeter,  DC  voltage.  Two  problems 
involved  the  measurement  of  ripple  voltage  and  negative  DC  voltage  using  the 
Simpson  volt-ohm-microammeter.  Model  269,  Series  11.  The  observer's  checksheet 
for  each  problem  was  divided  into  the  four  main  areas  of  (1)  initial  procedures, 

(2)  control  settings,  (3)  problem  solution,  and  (4)  safety.  Relevant  switches 
were  listed  under  the  first  two  areas  so  that  their  positions  could  be  checked-off. 
Each  of  the  main  areas  was  scored  with  a zero,  one,  or  two,  according  to  the 
criteria  shown  in  Table  1.  In  general,  a zero,  one,  or  two  was  assigned  for 
inadequate,  weak,  and  strong  performance,  respectively. 

Remedial  Shipboard  Training  Materials 

To  develop  training  corresponding  to  areas  of  potential  deficiencies,  the 
PPPs  covered  by  all  of  the  tests  were  grouped  and  matched  to  lesson  objectives 
so  that  each  PPP  item  was  included  in  at  least  one  lesson.  The  PPP  skill  items 
pertaining  to  the  operation  of  MTRE  and  use  of  test  equipment  tended  to  be  very 
general.  Therefore,  the  lessons  associated  with  the  MTRE  Preventive  Maintenance 
Test  and  the  Test  Equipment  Test  were  organized  around  the  test  problems.  The 
assignment  of  a lesson  was  based  on  noncompletion  of  performance  steps  anil 
inadequate  or  weak  performance.  On  the  other  hand,  the  PPP  knowledge  items 
addressed  by  the  DMS  problems  tended  to  be  more  detailed.  Therefore,  the  lessons 
associated  with  this  test  were  defined  primarily  in  terms  of  the  PPP  groupings 
and  the  assignment  of  a lesson,  which  was  based  on  wrong  test  answers,  cut  across 
several  different  problems. 

The  training  materials  were  developed  in  accordance  with  Navy  Ordinance 
documents  concerned  with  developing  and  preparing  FBM  Weapon  System  curricula 
(Note  5).  The  primary  type  of  material  was  the  trainee  guide  in  the  form  of  four 
self-study  workbooks.  Some  reading  materials  were  provided  in  the  workbooks, 
but,  in  other  cases,  reading  materials  were  assigned  in  various  technical  manuals. 
Training  aids  in  the  form  of  practice  troubleshooting  exercises,  audiovisual 
material,  and  a test-signal  generator  supplemented  the  self-study  materials. 

Review  of  Safely  Precautions 

This  self-study  workbook  was  a short,  programmed  instruction  booklet 
designed  to  provide  a review  of  general  personal  and  equipment  safety  precautions. 
It  was  assigned  as  remedial  Instruction  if  any  safety  violation  was  observed  U'.g., 
failure  to  remove  jewelry  or  watches  before  working  on  energized  equipment)  during 
the  Test  Equipment  Test  or  the  MTRE  performance  tests. 
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Table  1 


Performance  Scoring  Guide  for  the  Test  Equipment  Test 


Check.  Sheet 

Point  Requirements 

Scoring  Areas 

6 

1 

2 

Initial 

Procedures 

Assembled  < 80% 

Assembled  80%  of 

Assembled  all 

of  equipment 

equipment 

equipment 

OR 

AND 

AND 

Preset  < 80%  of 

Preset  80%  of 

Preset  all  controls 

controls 

controls 

OR 

AND 

AND 

Set  < 80%  of 

Set  > 80%  of 

Set  all  controls 

controls  after 

controls  after 

after  power  turn-on 

power  turn-on 

power  turn-on 

OR 

AND 

AND 

Required  exten- 

Minor  or  no 

No  assistance 

sive  assistance 

assistance 

Control 

Settings 

More  than  one 

One  error 

No  errors 

error 

OR 

OR 

AND 

Extensive 

Minor  assistance 

No  assistance 

assistance 

Problem 

Solution 

> * 10%  error 

< * 10%  error 

j:  1 1%  error 

OR 

OR 

AND 

Extensive 

Minor  assistance 

No  assistance 

assistance 

(one  prompt) 

(some  prompting) 

Safety 

More  than  one 

One  violation 

No  violations 

violation 

Troub leshooting  MTRE  Faults 

This  self-study  workbook  followed  the  typical  format  of  a trainee's  guide; 
that  is,  each  lesson  consisted  of  an  assignment  sheet  containing  study  assignments 
in  a technical  manual  and  a job  sheet  containing  self-test  exercises.  I lie- 
workbook  consisted  of  seven  lessons.  In  addition  to  the  usual  study  assignments 
and  self-test  items,  three  of  the  lessons  in  the  workbook  required  the  NT  to 
complete  a set  of  written  troubleshooting  problems.  A specific  lesson  was 
assigned  if  questions  across  different  DMS  problems  relating  to  the  lesson  were 
answered  incorrectly.  The  cut-off  score  or  criterion  for  assigning  each  lesson 
was  arbitrarily  set  at  70  percent  of  the  set  of  questions  involved.  In  addition, 
one  of  the  lessons  was  assigned  if  one  of  the  steps  (i.e.,  evaluation  of  mal- 
function) in  the  MTRE  Corrective  Maintenance  Test  exceeded  a 10-minute  time 
limit. 

MTRE  Maintenance  Procedures 


This  self-study  workbook  also  followed  the  typical  format  of  a trainee's 
guide.  However,  in  addition  to  the  usual  study  assignments  and  sclf-Lesi  items, 
each  lesson  included  an  exercise  that  required  the  MT  to  view  an  audiovisual 
module  and  to  complete  a physical  task  using  MTRE.  The  workbook  consisted  ol 
eight  Lessons  keyed  to  different  parts  of  the  MTRE  Prevei.tive  and  Corrective 
Maintenance  Tests.  A lesson  was  assigned  if  any  of  the  performance  steps  keyed 
to  that  lesson  were  done  incorrectly. 

Test  Equipment  Measurements 

This  self-study  workbook  contained  all  of  the  instructional  materials 
needed  to  complete  18  Lessons  covering  three  different  pieces  of  test  equipment: 
the  oscilloscope,  the  differential  voltmeter,  and  the  vol t-ohm-microammetcr . 

Three  of  the  18  lessons  were  introductions  to  each  of  the  three  pieces  of  tist 
equipment.  The  remaining  procedural  lessons  were  keyed  to  calibrations  or  measim  — 
ments  covered  in  the  test  equipment  test.  One  of  the  procedural  lessons  was 
assigned  if  an  MT's  performance  was  either  weak  or  inadequate  on  the  problem  to 
which  It  was  keyed.  If  any  procedural  lesson  on  a piece  of  equipment  was  assigned 
as  remedial  instruction,  the  associated  introductory  lesson  was  also  assignee. 

Exper imenta I Design 

The  experimental  design  shown  in  Table  2 was  used  to  evaluate  the  MT 
application  of  the  Personnel  Readiness  Training  Program.  As  depicted  in  the 
design,  three  experimental  groups,  each  consisting  of  four  submarine  MT  crews, 
were  used  in  the  evaluation:  a Control  Group,  a Diagnostic  Feedback  Croup,  and 

a Diagnostic  Feedback  + Training  Croup.  The  MT  crews  were  randomly  assigned 
to  one  of  the  groups.  Members  of  all  three  groups  were  given  diagnostic  tests 
and  then  were  retested  approximately  j months  later  using  the  same  tests. 

What  happened  to  the  individuals  during  the  interim  depended  upon  the  group  to 
which  they  were  assigned. 

Control  Croup 

Following  the  pretest,  members  of  tills  group  were  given  feedback  on  tlnu 
test  performance  in  terms  of  an  overall  percentage  score.  For  example,  an 
MT  was  told  that  bis  scores  were  b>  percent  for  the  MTRE  Preventive  Maintenniun 
Test,  <>8  percent  for  the  DMS  a Test,  and  'jto  percent  for  the  Test  Equipment  Tosi  . 
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Table  1 


(Experimental  Design  Used  in  the  MT  Application  of  the 
Personnel  Readiness  Training  Program 


Group 

Experimental 

Cond i t ions 

Gout  ro 1 

Pretest  with  non- 
specific feedback 

Patrol 

Posttest 

Di agiiost  ic  Feedback 

Pretest  with  specific 
feedback  on  weaknesses 

Pat  ro  1 

Post t est 

Diugnost ic  Feedback 
+ Training 

Pretest  with  specific 
feedback  on  weaknesses 
and  remedial  training 
ass igned 

Pa  t ro  1 

Special  Training 
Packages  Onboard 

Post t est 

Members  of  this  group  wore  not  provided  with  any  information  concerning  their 
specific  deficiencies,  but  they  were  not  given  suggestions  or  directions  as  to 
how  their  deficiencies  might  be  corrected. 

Diagnostic  Feedback  Group 

Following  the  pretest,  members  of  this  group  wore  provided  with  an  outline 
indicating  their  individual  weaknesses.  For  example,  if  an  NT  missed  any  of 
the  performance  steps  related  to  reloading  the  tape  cannister,  he  was  informed 
lie  was  weak  in  that  area.  Likewise,  if  an  NT  could  not  determine  phase  difference 
using,  an  oscilloscope,  lie  was  informed  he  was  weak  in  that  area.  As  with  tin 
Control  Croup,  members  of  this  group  were  nol  provided  with  any  information  as 
to  bow  tlii'ir  deficiencies  might  he  corrected. 

Pi, ignostic  Feedback  + Training  Croup 

Following  the  pretest,  members  of  this  group  were  given  the  same  type  ot 
feedback  as  was  given  to  those  in  the  Diagnostic  Feedback  Group;  that  is, 
list  of  individual  weaknesses.  In  addition,  they  were  assigned  specific  remeJi.  1 
training,  materials  covering,  their  individual  weaknesses.  For  example,  il  tin 
test  results  for  an  MT  indicated  that  lie  was  deficient  in  loading  tin  tape  canisi 
and  measuring  frequency  witli  an  oscilloscope,  he  was  assigned  the  self-instruction, 
lessons  that  covered  those  areas.  Members  ol  this  group  wore  asked  to  complete 
their  assigned  training,  at  times  convenient  for  them  during  t lie  i i subsequent 
patrol.  Fad i MT  was  also  supplied  with  a Progress  Checklist,  which  was  to  In 
initialed  by  his  supervisor  when  a lesson  was  completed.  flics,  I’rogiiss  Checklist 
were  similar  to  those  that  accompany  the  sell— study  workbooks  available  on  . 
request  basis  from  PTF.V.  A master  list  ol  training,  mat  or  in  Is  assigned  to  each 
MT  was  provided  to  the  Training  Officer,  or  the  heading,  Petty  01  t icer<  toi  tin- 
purposes  of  monitoring  training,  and  maintaining,  a record  as  each  MT  completed 
his  assigned  materials.  A meeting,  was  held  with  represcnl.it  ives  oi  < acli  ct  c\. 
assigned  to  this  group,  during  which  the  experimental  program  was  i -.plained  ana 
the  use  itf  tlte  training  materials  to  supplement  ongoing  shipboard  trainin'-  va 


encouraged.  Just  how  and  when  the  training  was  to  be  accomplished,  however, 
was  left  up  to  the  individual  submarines. 


Samp  le 

Missile  Technicians  from  12  Atlantic  Fleet  F15M  submarines  participated  in 
the  program  evaluation.  Although  four  submarines  were  assigned  to  each  group, 
the  size  of  the  MT  crew  varied  from  submarine  to  submarine.  Therefore,  the 
group  sizes  varied.  A total  of  77  MTs  participated  in  the  pretesting;  30  MTs 
were  assigned  to  the  Control  Group;  24,  to  the  Diagnostic  Feedback  Group;  and 
23,  to  the  Diagnostic  Feedback  + Training  Group.  Fourteen  MTs  were  unavailable 
for  the  posttesting  due  to  discharge,  transfer,  or  leave,  thereby  reducing  the 
Control  Group  to  26;  the  Diagnostic  Feedback  Group,  to  18;  and  the  Diagnostic 
Feedback  + Training  Group,  to  19.  The  distribution  of  pay  grades  and  the  percen- 
tage of  the  total  sample  each  represents  are  shown  in  Table  3. 


Table  3 


Percentage 

Distribution  of 

Pay  Grades  in 

the  Experimental 

Groups 

Pay  Grades 

D i agnos t ic 
Feedback  + 
Training  Group 
(N  = 19) 

Diagnostic 
Feedback 
Group 
(N  = 18) 

Control 
Group 
( N = 26) 

Total 
Sample 
(N  = 63) 

E-4 

53 

28 

35 

38 

E-  3 

37 

56 

46 

46 

E-6 

10 

5 

8 

8 

E-7/F.-8 

0 

11 

1 1 

8 

Procedures 

Fleet  testing  was  conducted  by  the  staffs  of  the  Personnel  and  Training 
['.valuation  Program  (PTEP)  Central  Test  Site  Detachments  located  at  Charleston 
anti  New  London.  The  PTEP  team  members  were  trained  as  test  proctors  by  members 
of  the  research  staff  during  pilot  evaluation  of  the  tests.  The  specially 
developed  MTRE  performance  tests,  the  test  equipment  test,  and  the  DMS  tosi  were 
administered  in  place  of  the  usual  testing  carried  out  by  PTEP.  I'hc  time  to 
carry  out  PTEP  testing  was  increased  from  l to  2 davs  because  individual lv  admin- 
istered performance  tests  require  more  time  than  the  group  written  tests  normally 
administered  bv  PTEP. 

A test  ing  schedule  for  each  MT  was  arranged  at  t lie  start  of  the  2-day  test  ing 
period.  Ibe  testing  usually  started  out  with  a group  administration  ol  three 
DMS  problems  followed  by  the  hands-on  performance  tests.  Only  those  MTs  who 


had  responsibility  for  M'L’RE  operation  and  maintenance  on  their  last  patrol  were 
Ivon  the  MTRK  Corrective  Maintenance  Test.  The  remaining  MTs  were  given  the 
.TIRE  Preventive  Maintenance  Test.  All  M'l's  were  given  the  Test  Equipment  Test. 
Each  MT  was  told  that  the  testing  program  was  experimental  in  nature  and  would 
take  tlie  place  of  the  usual  PTEP  testing. 

The  MTs  were  not  informed  that  there  were  three  different  groups  involved 
in  the  experimental  program.  They  were  told,  iiowever,  that  everyone  would  he 
retested  following  their  next  patrol.  At  posttest,  the  MTs  completed  a short 
biographical  questionnaire.  The  MTRE  performance  tests  and  the  test  equipment 
test  administered  at  posttest  were  the  same  as  those  administered  at  pretest. 

Of  the  three  DMS  problems  administered  at  posttest,  one  was  exactly  the  same  as 
that  administered  at  pretest  so  that  a rough  estimate  of  reliability  could  be 
obtained.  The  other  two  problems  were  different  to  eliminate  the  possibility 
of  remembering  the  problem  solution,  which  is  always  provided  by  the  nature  of 
the  test.  All  of  the  PPP  items  covered  in  the  DMS  pretest  problems  were  covered 
in  the  posttest  problems,  and  the  total  number  of  questions  was  the  same  at  pretc 
and  post  test. 
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It  is  important  that  a performance  test  yield  similar  results  when  used  by 


different  raters.  Consequently,  during  the  pretesting  of  27  M'ls,  the  object  ivit> 
of  the  MTRF  performance  tests  was  evaluated  by  using  two  raters  from  the  research 
staff  individually  paired  with  PTF.P  raters.  A total  of  79  problems  was  observed 
and  separately  recorded  by  the  different  pairs  of  raters.  Those  steps  that  an 
MT  did  not  attempt  to  complete  were  not  included  in  the  evaluation  since  this 
would  have  resulted  in  an  inflated  amount  of  agreement  between  raters.  For  the 
2>  M'l's  observed  on  the  three  preventive  maintenance  problems,  the  percentage  ot 
agreement  between  the  different  pairs  of  raters  was  92,  99,  and  94  percent  for 
the  tape  tault,  digital  multimeter,  and  tape  reader  problems,  respectively . 

Only  2 of  the  27  MTs  were  given  the  corrective  maintenance  Lest,  and  there  was 
9(>  percent  agreement  on  the  diagnostic  steps  observed  in  these  problems. 

When  test  performance  is  related  to  whether  or  not  a cut-off  score  is  met , 
it  is  classified  as  a criterion-referenced  test,  and  most  standard  measures  ot 
reliability  and  validity  cannot  be  list'd  (Popharn  & llusek,  19t>9).  In  the  present 
program,  where  the  purpose  is  to  diagnose  deficiencies  anti  assign  training  it' 
remedy  those  deficiencies,  test  reliability  can  be  viewed  in  terms  of  the  con- 
sistency of  the  diagnostic  decisions  that  would  be  made  at  two  different  point.-, 
in  time.  In  this  program,  the  Control  Croup  received  no  special  treatment 
(i.e.,  no  specific  feedback  and/or  training  materials).  Thus,  although  up  t. 

:>  months  passed  between  pretesting  and  posttesting,  this  group  provided  data 
to  estimate  test-retest  reliability.  For  the  three  preventive  maintenanci 
problems,  there  was  agreement  in  diagnostic  decisions  from  pretest  to  posttest 
of  8.)  percent  for  the  tape  fault  problem,  74  percent  for  l lie  digital  mult  imet  ci 
problem,  and  74  percent  for  the  tape  reader  problem.  For  the  two  corrective 
maintenance  problems,  there  was  7b  percent  agreement. 

the  MTRF.  performance  tests  have  high  face  validity  because  the  MTs  have  t. 
operate  MTRF..  The  purpose  of  the  MTRF.  performance  tests  was  to  determine  it 
preventive  maintenance  procedures  and  basic  troubleshooting  techniques  could 
be  performed  properly.  The  tests  have  high  content  validity  since  the  perfor- 
mance steps  are  replicates  of  procedural  steps  found  in  maintenance  documents 
for  this  equipment. 

I'revcnt  ive  Ma  i n t en.ince 

For  the  three  preventive  maintenance  problems,  .ill  groups  performed  poorl\ 
on  the  pretest  and  all  improved  on  the  posttest.  However,  there  was  little  diffe- 
rence between  the  groups.  This  is  shown  in  Figures  1,  2,  and  i,  in  terms  ot  t hi 
percent  of  performance  steps  completed.  Analyses  ot  variance  with  repeated, 
measures  on  the  prel cst /post  lost  variable  indicated  that  the  average  improvement 
ol  19  percent  on  the  tape  splicing  and  reloading  procedure,  10  percent  on  the 
Pig  it.)  I Multimeter  Floetrieal  Adjustment  Check,  and  1 > percent  on  the  1 ape  Kc.uh  t 
Flectrlc.il  Adjustment  Check  were  statistically  significant  ( F(  1 , >1)  = 2>.  >(-, 

£ .01;  F ( 1 , 47)  17.04,  £ • .01;  F(l,  49)  = 19.14,  £ .01;  respectively). 
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Correct  ive  Ma  intenanee 


Since  only  nine  MTs  received  these  two  problems,  no  statistical  analyse 
were  perfi>rmod  on  the  hands-on  troubleshooting  data.  The  main  measure  obtained 
in  tin-  troubleshooting  problems  was  the  solut  ion/tionso  1 u t ion  of  the  problem. 

Table  4 shows  the  number  of  MTs  who  solved  zero,  one,  or  two  of  the  trouble- 
shooting problems  at  pretest  and  posttest.  Performance  was  collapsed  over 
groups  because  there  was  little  difference  in  performance  between  the  groups. 
Approximately  half  of  the  MTs  who  had  primary  responsibility  for  MTKK  operation 
and  maintenance  on  their  last  patrol  could  not  solve  the  two  problems  at  either 
pretest  or  posttest.  There  was  little  change  in  performance  from  pretest  to 
posttest.  The  average  number  of  test  measurements  made  by  the  MTs  varied  greatly. 
They  made  either  very  few  measurements  or  a great  many  measurements  in  unsuccess- 
ful attempts  to  solve  a troubleshooting  problem. 


Tab  I e 4 

Number  of  MTs  Solving  Troubleshooting  Problems  at  Pretest  and  Post  lest 


Problems  Solved 

I'ond  i t i on 

None 

One 

Two 

I’ret  est 

4 

3 

- 

Post  test 

5 

l 

3 

Decision  Measurement  System 

(DMS)  Test 

Ke I iab i I i t y and  Va 1 id i l y 

The  three  DMS  problems  administered  at  pretest  and  pastiest  were  net  all 
the  same:  two  posttest  problems  were  changed  to  eliminate  the  possibility  ot 

remembering  the  problem  solution.  'Therefore,  the  reliability  of  this  trouble- 
shooting test  could  not  be  estimated  in  terms  of  the  consistency  of  the  diaguosti 
decisions.  The  possibility  of  remembering  the  problem  solution  argues  against 
estimating  reliability  as  a correlation  between  pretest  and  posttest  scores  t>n 
l be  Control  Crimp  on  the  one  problem  repeated;  however,  there  was  a period  ol 
approximately  r>  months  between  pretest  and  posttest  during  which  correct  select  io 
discovered  through  t rial-aud-error  at  pretest  might  he  forgotten.  Tims,  a cor- 
relation mav  provide  a very  rough  indication  of  reliability.  Keeping  tin  above 
reservations  in  mind,  the  Pearson  produe  t -moment  correlation  was  . it)  between 
pretest  and  post  test  scores  on  tile  one  repeated  problem. 


The  validity  of  the  1)MS  test  is  based  on  the  fact  that  the  paper-and-peiu  i l 
problems  were  developed  by  inserting  faults  in  MTRE,  following  accepted  trouble- 
shooting procedures  to  isolate  the  faults,  and  noting  the  conditions  at  each 
major  step  throughout  the  troubleshooting  procedure.  The  extensive  use  of 
display  pictorials,  machine  printouts,  and  waveform  photographs  gave  the  Mi  lU 
troubleshooting  problems  both  high  face  and  content  validity.  It  should  be  noted, 
however,  that  performance  on  the  paper-and-pencil  DMS  problems  has  not  been  com- 
pared with  hands-on  performance  on  MTRE.  Therefore,  the  relationship  between 
simulated  troubleshooting  performance  and  actual  troubleshooting  is  not  known. 

Troubleshoot ing  Problems 

There  were  no  significant  effects  found  for  any  of  the  variables  when  th 
total  DMS  score  over  all  three  problems  was  analyzed.  Table  5 shows  that  there 
was  little  change  from  pretest  to  posttest  and  no  differences  between  the  three 
groups  in  terms  of  the  total  points  attained  over  the  three  problems. 

Table  5 

Mean  Total  Points  Attained  on  the  Three  DMS  Problems 
at  Pretest  and  Posttest  for  Three  Groups 


Groups 


D iagnos t i c 
Feedback  + 

Condition  Training  Group 


Diagnost ic 
Feedback 
Group 


Contro 1 
Group 


Pretest 


101 


207 


195 


Post  test 


198 


210 


209 


Note. — Maximum  score  = 252. 


Test  Equipment  Test 

Oh j ec 1 1 v 1 ' v.  Reliability,  and  Validity 

Objectivity  of  the  Test  Equipment  lest  was  evaluated  in  the  same  way  as 
the  MTRE  performance  tests;  that  is,  two  raters  from  the  research  staff  were 
individually  paired  with  PTF.P  raters  during  the  pretesting  of  20  MTs.  A total 
of  160  problems  were  observed  and  separately  recorded  by  the  different  pairs 
of  raters.  If,  In  any  of  the  three  scoring  areas  (e.g.,  initial  procedures, 
control  settings,  or  problem  solution),  an  MT  did  not  attempt  to  complete  t he 
steps,  that  area  was  not  included  in  the  evaluation  since  this  would  have  resulted, 
in  an  Inflated  amount  of  agreement  between  raters.  The  area  of  safety  was  included 
in  the  evaluation  only  if  one  or  more  of  the  other  three  scoring  areas  were 
attempted.  For  the  three  problems  using  the  oscilloscope,  there  was  81  percent 


agreement  between  raters;  for  the  three  problems  using  the  d i f forent ia I volt- 
meter, 81  percent  agreement;  for  the  two  problems  using  the  vo I t -ohm-mi e roammei oi , 
6 7 percent  agreement. 


Using  t hi'  same  procedure  followed  with  the  MTRK  Preventive  Maintenance  lest, 
the  reliability  of  the  Test  Equipment  Test  was  estimated  in  terms  of  t he  consistency 
ol  the  diagnostic  decisions.  For  the  oscilloscope,  there  was  81  percent  agreement 
in  the  diagnostic  decisions  from  pretest  to  posttest;  for  the  differential  voltmeter, 
60  percent  agreement;  for  the  volt-ohm-microammeter,  77  percent  agreement.  For 
observing  safety  precautions  over  all  test  equipment  problems,  there  was  Mo  percent 
.agreement  in  the  diagnostic  decisions  from  pretest  to  posttest. 

The  Test  Equipment  Test  has  high  face  validity  because,  during,  the  test, 
the  Mis  have  to  make  calibration  adjustments  and  measurements  using  the  equipments 
in  the  same  manner  as  on  the  job.  The  test  has  high  content  validity  since  the 
steps  covered  by  the  scoring  areas  are  taken  from  the  equipment  technical  manuals. 

The  validity  of  the  individual  problems  was  further  assured  bv  having,  several 
experienced  MTs  at  the  two  MTRP.  equipment  facilities  "walk  through"  the  test 
problems.  (One  of  the  original  oscilloscope  problems  was  eliminated  alter  this 
test  tryout  when  iL  was  found  that  MTs  do  not  usually  make  the  measurement  it 
requ i red . ) 

Test  Kquipment  Use 

For  all  ti'St  equipments,  all  groups  performed  poorly  on  the  pretest,  and 
all  improved  on  the  posttest.  Thus,  there  was  little  difference  between  the 
groups.  This  is  shown  in  Figures  4,  5,  and  6,  in  terms  of  the  percent  of  possible 
points  that  could  have  been  attained  over  the  scoring  areas  of  initial  procedures, 
control  settings,  and  problem  solution.  Analyses  of  variance  with  repeated 
measures  on  tlu1  pr  test/posttest  variable  indicated  that  the  average  improvement 
of  11  percent  on  the  oscilloscope,  19  percent  on  the  differential  voltmeter, 
and  lb  percent  on  the  volt-ohm-microammeter,  were  statistically  significant 
(F(l,  60)  = 17.48,  £ ■ .01;  F(l,  bO)  = 19.22,  £ < .01;  F( 1 , 60)  = 17.81,  £ .01; 

respectively).  The  average  improvement  of  15  percent  on  observing  safety  pre- 
cautions over  all  problems  was  also  statistically  significant  (F( 1 , bO)  = 17.81, 
p .01).  No  other  effects  wore  found  to  be  significant  in  anv  of  those  analyses. 

As  with  the  MTRK  performance  tests,  an  additional  set  of  analyses  was  performed 
to  see  if  the  average  improvement  from  pretest  to  posttest  was  due  to  those  MTs 
in  the  sample  who  did  noL  attempt  one  or  more  of  the  problems  at  protest  hut  did 
try  the  same  problem,  or  problems,  at  posttest.  Removal  of  these  individuals 
did  not  change  the  results. 
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Figure  6.  Performance  Using  the  Simpson  Model  269,  Series  II,  Volt-ohm-Microammeter 


DISCUSSION 


Performance  Deficiencies 


The  pretest  results  showed  that  there  were  serious  deficiencies  in  tasks 
related  to  the  maintenance  of  the  MTRE.  In  fact,  the  initial  performance  levels 
were  so  low  that  almost  all  MTs  were  assigned  the  entire  remedial  training 
package.  Prior  to  testing,  it  had  been  anticipated  that  a considerable  number  of 
MTs  would  require  little  or  no  remedial  training.  On  preventive  maintenance 
tasks,  the  overall  percentage  of  steps  completed  correctly  was  only  57  percent 
(Figures  1,  2,  and  3).  This  level  of  performance  is  especially  low  when  it 
is  considered  that  the  appropriate  maintenance  documentation  was  in  the  hands 
of  the  MT  during  testing.  An  examination  of  the  effect  of  previous  school 
experience  on  the  performance  of  these  tasks  revealed  that  a significantly 
greater  percentage  of  steps  were  completed  (80  percent)  by  the  19  MTs  who  bail 
attended  the  MTRE  Maintenance  Course  than  by  the  35  MTs  who  had  not  (48  percent) 
(£  (52)  = 3.76,  £ < .01).  The  largest  contribution  to  the  poor  performance  of 
the  MTs  without  MTRE  classroom  experience  was  made  by  newly  rated  MTs,  several 
of  whom  could  not  even  find  the  applicable  procedures  in  the  documentation 
available  during  the  testing.  Perfect  performance  was  a rare  event,  even  among 
those  who  had  classroom  training  on  MTRE.  It  should  be  noted  that  the  time 
between  completion  of  the  MTRE  Maintenance  Course  and  testing  ranged  from  4 to 
29  months.  The  maintenance  skills  of  the  MTs  given  the  MTRE  Preventive 
Maintenance  Test  might  be  expected  to  decrease  during  this  period  because  they 
had  not  had  recent  responsibility  for  MTRE  and,  therefore,  had  no  opportunity 
to  practice  these  skills. 

As  indicated  previously,  the  few  MTs  who  had  responsibility  for  MTRE  on  the 
patrol  previous  to  the  testing  were  given  the  MTRE  Corrective  Maintenance  Test. 
Their  troubleshooting  performance  showed  serious  deficiencies.  Performance 
on  the  troubleshooting  problem  of  medium  difficulty  is  of  most  interest  because 
it  uses  a prefaulted  module  and  is  very  similar  to  the  problems  given  to  students 
in  the  MTRE  Maintenance  Course.  Four  out  of  the  nine  MTs  who  had  responsibility 
for  MTRE  on  their  last  patrol  could  not  solve  this  problem  at  pretest.  Surpris- 
ingly, three  of  these  nine  MTs  had  not  attended  the  MTRE  Maintenance  Courst  . 

Yet,  one  of  these  three  MTs  was  able  to  solve  the  problem  at  pretest,  while  three 
out  of  six  MTs  who  had  attended  the  course  could  not  solve  the  problem.  Phis 
seems  to  suggest  that  classroom  training  on  MERE  may  not  be  strongly  related 
to  troubleshooting  performance.  Performance  on  the  simulated  troubleshooting 
problems,  given  to  all  of  the  MTs,  lends  support  to  this  suggestion.  There  was 
little  difference  in  the  mean  total  number  of  points  attained  on  this  pretest 
between  MTs  who  had  MTRE  classroom  training  (210)  and  those  who  had  not  tl9b). 
Although  the  MTRE  Maintenance  Course  may  be  fundamental  to  performing  preventive 
maintenance,  it  appears  that  other  factors  may  be  of  equal  importance  in  devi  lop- 
ing troubleshooting  skills.  It  is  probably  safe  to  conclude  that  one  of  these 
factors  is  practice  in  performing  corrective  maintenance,  which  does  not  occur 
frequently  on  MTRE  because  of  its  exceptional  reliability. 

Serious  performance  deficiencies  were  also  found  in  the  use  of  test  equipment 
While  the  deficiencies  encountered  in  the  use  of  the  differential  voltmeter  and 
the  vo 1 t-ohm-mic roamme ter  were  less  serious  than  those  encountered  in  the  use 
of  the  oscilloscope,  performance  was  still  not  adequate  to  support  maintenance. 
Previous  school  experience  was  related  to  more  proficient  test  equipment  use, 
just  as  it  was  related  to  better  performance  on  MERE  preventive  maintenance  task 
Over  all  three  pieces  of  test  equipment,  a significantly  greater  percentag*  of 
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points  was  attained  (46  percent)  by  MTs  who  had  attended  a test  equipment  course 
than  by  those  who  had  not  (31  percent)  (t  (61)  = 1.81,  £ ■ .03).  Only  10  of 
the  63  MTs  tested  had  the  advantage  of  classroom  experience  devoted  solely  to 
test  equipment  use.  For  those  10,  the  time  since  completion  of  the  course  ranged 
from  1 to  to  96  months.  Proficiency  might  be  expected  to  decrease  over  at  least 
the  longer  time  periods  because  of  intermittent  practice  opportunities,  especially 
in  the  case  of  the  most  complicated  piece  of  test  equipment,  the  oscilloscope. 

In  regard  to  the  oscilloscope,  not  only  did  the  MTs  have  difficulty  in  making 
calibration  adjustments  and  measurements,  but  a majority  of  them  had  trouble 
with  the  basic  turn-on  procedures.  In  fact,  many  MTs  could  not  even  setup  the 
equipment.  A comment  made  by  many  MTs  during  testing  was  that  the  oscilloscope 
is  usually  already  setup  and  calibrated,  and,  if  it  is  not,  an  Electronic 
Technician  (ET)  can  be  asked  to  do  it.  However,  the  poor  performance  exhibited 
by  the  MTs  on  use  of  test  equipment  prompted  PTEP  to  examine  a selected  sample 
of  Sonar  Technicians  and  Electronic  Technicians  using  an  adaptation  of  the  lest 
Equipment  Test.  The  results  for  these  two  samples  were  similar  to  those  for  the 
MTs  (PTEP  Evaluation  3-76,  Note  6)  and  are  in  line  with  previous  research  (e.g., 
Abrams  6.  Pickering,  1962;  Steadman  6 Harrigan,  1971;  Steinemann,  Coady,  Harrigan, 
Matlock,  & Steadman,  1969),  which  has  repeatedly  shown  serious  problems  in  the 
use  of  test  equipment.  The  MT's  notion  that  he  can  rely  on  the  ET  for  help  in 
using  the  oscilloscope  is  not  borne  out  by  this  data. 

The  results  from  the  12  MT  crews  tested  showed  that  all  groups  improved 
significantly.  Neither  feedback  on  individual  weaknesses  nor  feedback  plus 
specifically  assigned  remedial  training  improved  job  performance  over  that  shown 
by  the  Control  Group.  Therefore,  the  improvement  that  did  take  place  for  the 
three  groups  was  probably  due  to  experience  gained  in  the  testing  situation. 

Absence  of  Training  and  Feedback  Effects 

There  was  no  training  effect  because  most  of  the  19  MTs  in  the  Training 
Group  did  not  complete  their  assigned  lessons.  This  was  confirmed  from  both 
self-reports  during  posttesting  and  from  the  Progress  Checklists.  These 
checklists  were  returned  by  only  two  of  the  four  submarines  in  the  Training 
Group.  Iff  the  two  sets  of  Progress  Checklists  returned,  only  one  contained 
supervisors'  initials  verifying  the  completion  of  lessons  and  was  accompanied 
by  a signed  "Training  Attendance  and  Critique  Record,"  which  usually  accompanies 
tht  completion  of  self-study  workbooks  obtained  from  PTEP.  As  a consequence, 
only  five  MTs  could  be  identified  as  having  completed  their  assigned  lessons. 

To  get  an  idea  of  whether  or  not  the  training  materials  might  have  im- 
proved hands-on  performance  if  they  had  been  more  widely  used,  the  five  MTs 
who  completed  their  lessons  were  singled  out  for  examination.  The  performance 
of  each  of  these  MTs  was  compared  to  that  of  similar  MTs  from  the  other  iwo 
groups;  tiiat  is,  members  of  the  Control  Group  and  Feedback  Group  who  would 
have  been  assigned,  at  the  minimum,  the  same  lessons  as  each  of  the  five 
MTs  were  selected  to  form  a yoked  suhsample.  It  should  be  noted  that  the  five 
Diagnostic  Feedback  + Training  Group  MTs  who  had  completed  their  lessens  had 
not  been  assigned  training  in  all  areas.  Table  6 shows  t lie  voked  subsample 
compared  in  terms  of  the  percentage  of  the  points  or  steps  that  were  missed  at 
pretest  but  were  attained  at  post  test.  For  example,  if  at  pretest  five  steps 
of  the  Digital  Multimeter  Electrieal  Adjustment  Check  were  either  incorrect  ei 
not  completed,  and  four  of  these  steps  were  completed  correct lv,  at  posttest,  then 
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the  "percent  improvement"  was  80  percent.  The  Training  Group  showed  more  improvt 
ment  than  the  other  groups  in  all  but  one  case.  While  these  comparisons  are  based 
on  extremely  small  subsamples,  they  do  suggest  that,  had  training  been  completed, 
it  would  have  improved  performance.  An  independent  study,  in  which  10  enlisted 
personnel  of  a variety  of  electronic  ratings  were  tested  ashore  and  then  given 
the  test  equipment  training,  confirmed  the  t».end  found  in  the  yoked  subsample. 

The  results  for  the  independent  study  are  summarized  in  Table  7.  There  was 
highly  significant  improvement  in  performance  after  completion  of  training  on 
the  three  pieces  of  test  equipment — at  posttest,  6 out  of  10  attained  over  90 
percent  of  the  possible  points  on  the  oscilloscope;  6 out  of  10,  over  90  percent 
on  the  differential  voltmeter;  and  9 out  of  10,  over  90  percent  on  the  volt-ohm- 
microammeter  (_F(1,  18)  = 278.0,  50.5,  and  58.2,  respectively,  all  £ • .001). 

Table  6 

Percent  of  Possible  Improvement  in  Hands-on  Performance  for  a Yoked  Subsample 


Experimental  Groups 

Test 

Diagnost ic 
Feedback  + 
Training  Group 

Diagnost Ic 
Feedback 
Group 

Contro 1 
Group 

Oscilloscope3 

80 

60 

28 

Test 
Equ  i p 

Differential  Voltmeter3 

86 

37 

>0 

ment 

Vol  t -Ohm-Mi croamme ter*’ 

100 

0 

4 4 

Tape  Splice/Canister  Load'* 

91 

0 

2 0 

MTRE 

Digital  Multimeter  Check*" 

3 i 

50 

7 b 

Note. — No  training  was  assigned 

to  this  subsample 

as  a resu It  of  t h< 

MTRF 

( nvo 1 v i ng 

th 

e tape 

reader 

= 5 

for 

al 

1 

three 

groups . 

= 3 

for 

al 

1 

t hree 

groups . 

^ 2 
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al 

1 
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Table  7 

Performance  of  Ten  Navy  Technicians  on  the  Test  Equipment  Test 
Before  and  After  Training 


Percent  Correct 

Use i 1 loscope 

1)  i f f erent  ial 
Vo  1 1 meter 

Vo  1 1 -Ohm 
Microammet er 

before  Training 

5 

22 

29 

After  Training 

90 

91 

94 

Although  the  Feedback  + Training  Group  was  given  feedback  and  specifically 
assigned  remedial  material,  they  still  did  not  improve  in  performance — mainly 
because  the  training  was  not  completed.  Given  this  fact,  it  is  not  surprising 
to  find  that  feedback  alone  had  no  effect.  The  Sonar  Technician  application 
(Winchell,  Panell  i»  Pickering,  1976)  also  found  that  feedback  alone  had  no  effect. 
It  was  suggested  that  it  may  have  been  too  difficult  for  the  Sonar  Technicians 
to  extract  the  needed  information  from  the  available  documents.  While  this  could 
possibly  be  true  in  the  present  application  for  the  test  equipment  information, 
it  is  obviously  not  the  case  for  the  MTRF.  information,  since  detailed  step-by-step 
procedures  are  listed  in  the  official  documents.  The  reasons  for  not  using,  feed- 
back are  probably  the  same  as  those  for  not  using  training. 

Comments  on  the  Nonuse  of  Training 

There  may  be  a number  of  reasons  why  the  assigned  training  was  not  used  by 
a majority  of  MTs  in  the  Training  Group  to  supplement  their  ongoing  shipboard 
training.  First  of  all,  there  may  have  been  some  features  of  the  training 
materials  that  contributed  to  their  nonuse.  For  example,  the  individual  lessons 
of  a training  package , which  ranged  from  approximately  1 to  2 hours,  were  much 
longer  than  the  '..-hour  lessons  of  the  Sonar  Technician  application  in  which 
training  was  found  to  be  used  (Winchell,  Panel!  & Pickering,  I97(>).  In  add  i t ion, 
testing  took  place  on  several  different  pieces  of  equipment  rather  than  just 
one  piece,  as  in  the  Sonar  Technician  application.  When  deficiencies  were  found, 

1 1 iese  features  resulted  in  the  assignment  of  lengthv  training  packages  covet  iug 
many  dill erent  areas . 

Personnel  assignment  practices,  peculiar  to  the  operation  and  maintenance 
of  MTRF,  probably  had  a stronger  influence  on  the  nonuse  of  training,  rnatct  ials 
than  lesson  length  or  training  package  size.  In  particular,  only  one  IT  is 
us  igned  the  responsibility  for  MTRF,  on  a given  patrol.  That  Ml  is  not  only  tin 


onlv  one  who  has  the  opportunity  to  practice  his  maintenance  skills,  but  lie  is 
a 1 so  the  only  one  who  is  likely  to  feel  that  those  skills  are  necessary  to  iiis 
job.  Moreover,  a given  MT  usually  Inis  responsibility  for  MTRF  only  once  during 


his  tour  on  an  FBM  submarine.  This  probably  serves  to  reduce  Itu'  perceived 
importance  and  relevance  of  the  training  for  those  MTs  not  currently  assigned 
responsibility  for  MTRE. 

Another  important  reason  for  nonuse  of  training  materials  is  that  MTRE 
is  an  exceptionally  reliable  piece  of  equipment.  Because  MTRE  operates  without 
failure  for  extended  periods  of  time,  studying  material  related  to  MTRE  is  likelv 
to  appear  to  be  unprofitable.  Unfortunately,  the  exceptional  reliability  of  Mi  Rl 
results  in  little  opportunity  to  practice  maintenance  skills,  and  this  sets  the 
stage  for  the  deterioration  of  these  skills. 

Implementation  Costs 

The  estimated  cost  of  reproducing  one  complete  set  of  training  materials, 
less  the  test  signal  generator,  is  $196.85.  The  amount  of  $195.74  is  a one-time 
cost  because  all  of  the  materials  are  reuseable,  with  the  exception  of  the  first 
self-study  workbook.  The  prototype  test  signal  generators  cost  $1350.00  each 
in  an  eight-item  lot,  which  includes  one-time  developmental  costs.  The  price 
of  this  equipment  can  be  expected  to  decrease  with  additional  orders,  especially 
if  the  lot  sizes  are  larger.  A breakdown  of  the  estimated  costs  by  training 
package  are  shown  in  Table  8.  Since  only  a limited  number  of  MTs  would  be 
tested  under  the  recommendations  that  follow,  the  time  requirements  for  MIRE 
equipment,  test  equipment,  and  a test  proctor  are  minimal.  The  hands-on  perfor- 
mance tests  can  be  completed  in  less  than  2‘s  hours,  and  testing  could  be  easily 
conducted  by  PTEP  during  the  routine  PTEP  testing. 


Table  8 

Estimated  Cost  of  Training  Materials 


Self-Study  Workbook 

Components 

Estimated  Cost 

Review  of  Safety  Precautions 

19  Printed  Pages 

$1.10 

Troubleshooting  MTRE  Faults 

60  Printed  Pages 

$2.75 

Simul a ted  Troubleshoot ing 
Exerc i ses 

$0.00“ 

MTRE  Maintenence  Procedures 

42  Printed  Pages 

$2.00 

238  Slides 

$75.00 

8 Audio  Cassettes 

$84.00 

1 Tape  Player 

$2b. 00 

Test  Equipment  Measurements 

233  Printed  Pages 

$6.00 

1 Test  Signal  Generator 

$1350.00 

Available  through  PTEP 


CONCLUSIONS 


It  is  clear  from  the  present  application  of  a diagnostic  testing  and  shipboard 
training  program  that  there  are  serious  deficiencies  in  tasks  related  to  the 
maintenance  of  MTRE.  It  is  also  clear  that  the  MT  and  those  responsible  for 
his  training  must  be  convinced  that  a problem  exists  that  can  be  helped  by 
training.  The  existing  MTRE  Maintenance  Course  may  be  all  that  is  needed  to 
produce  an  MT  who  can  accomplish  MTRE  preventive  maintenance  tasks.  However, 

MTRE  classroom  training  is  not  likely  to  be  helpful  unless  received  in  the  off- 
crew  period  just  prior  to  assignment  of  MTRE  responsibility.  The  same  probably 
holds  true  for  the  test  equipment  courses. 

It  is  recognized  that  not  all  MTs  will  be  sent  to  the  appropriate  schools 
just  prior  to  being  assigned  MTRE  responsibility.  Some  will  already  have  attended 
these  schools.  The  present  testing-and-training  program  can  play  an  important 
role  in  identifying  performance  deficiencies  in  those  who  have  received  school 
training  in  the  past.  By  testing  those  MTs  at  the  beginning  of  the  off-crew 
period  just  prior  to  their  assignment  of  MTRE  responsibility,  any  performance 
deficiencies  can  be  revealed.  These  deficiencies  can  then  be  corrected  through 
the  use  of  the  self-study  workbooks  of  the  present  program. 
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RECOMMENDATIONS 

To  improve  the  proficiency  level  of  the  MT  designated  to  have  responsibility 
for  operating  and  maintaining  MTRE,  it  is  recommended  that: 

1.  If  the  designated  MT  has  not  already  attended  the  MTRE  maintenance  and 
test  equipment  courses,  he  should  attend  these  courses  during  the  off-crew  period 
immediately  preceding  the  patrol  on  which  he  will  assume  MTRE  responsibility. 

2.  If  the  designated  MT  has  attended  the  MTRE  maintenance  and  test  equipment 
courses,  he  should  be  given  the  MTRE  diagnostic  test  at  the  beginning  of  the 
off-crew  period  immediately  preceding  the  patrol  on  which  he  will  assume  MTRE 
responsibility.  He  should  then  be  assigned  the  appropriate  remedial  training 
package  and  be  retested  at  the  completion  of  training. 
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